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ABSTRACT 

Context. Absorption by molecules in the Earth’s atmosphere strongly affects ground-based astronomical observations. The resulting 
absorption line strength and shape depend on the highly variable physical state of the atmosphere, i.e. pressure, temperature, and 
mixing ratio of the different molecules involved. Usually, supplementary observations of so-called telluric standard stars (TSS) are 
needed to correct for this effect, which is expensive in terms of telescope time. We have developed the software package molecfit 
to provide synthetic transmission spectra based on parameters obtained by fitting narrow ranges of the observed spectra of scientific 
objects. These spectra are calculated by means of the radiative transfer code LBLRTM and an atmospheric model. In this way, the 
telluric absorption correction for suitable objects can be performed without any additional calibration observations of TSS. 

Aims. We evaluate the quality of the telluric absorption correction using molecfit with a set of archival ESO-VLT/X-Shooter visible 
and near-infrared spectra. 

Methods. Thanks to the wavelength coverage from the U to the K band, X-Shooter is well suited to investigate the quality of the 
telluric absorption correction with respect to the observing conditions, the instrumental set-up, input parameters of the code, the 
signal-to-noise of the input spectrum, and the atmospheric profiles. These investigations are based on two figures of merit, /„g and 
7 res , that describe the systematic offsets and the remaining small-scale residuals of the corrections. We also compare the quality of the 
telluric absorption correction achieved with molecfit to the classical method based on a telluric standard star. 

Results. The evaluation of the telluric correction with molecfit shows a convincing removal of atmospheric absorption features. 
The comparison with the classical method reveals that molecfit performs better because it is not prone to the bad continuum 
reconstruction, noise, and intrinsic spectral features introduced by the telluric standard star. 

Conclusions. Fitted synthetic transmission spectra are an excellent alternative to the correction based on telluric standard stars. 
Moreover, molecfit offers wide flexibility for adaption to various instruments and observing sites. 

Key words. Radiative transfer - Atmospheric effects - Instrumentation: spectrographs - Methods: data analysis - Methods: numer¬ 
ical - Techniques: spectroscopic 


1. Introduction 

Ground-based observations are naturally affected by various 
physical processes in the Earth’s atmosphere, in particular scat¬ 
tering and absorption. The dynamics of the weather conditions, 
seasonal effects, or climate change lead to variabilities in tem¬ 
perature, pressure, and the chemical composition on time scales 
from seconds to decades, making the absorption correction a de¬ 
manding matter. Thus, any data calibration usually needs sup¬ 
plementary calibration frames. However, this approach is very 
expensive in terms of telescope time, because these data should 
be taken directly before or after the science target. 


http://www.eso.org/sci/software/pipelines/skytools/ 


This applies particularly to the correction arising from 
molecular absorption in the Earth’s atmosphere. The required 
supplementary calibration frames are observations of telluric 
standard stars (hereafter TSS). The TSS are usually bright hot 
stars (e.g. B-type), that show only a few, well known intrinsic 
spectral features. The TSS have to be observed at a similar air- 
mass, time, and line of sight as the science target to probe the 
same atmosphere column. 

Within the framework of the in-kind contribution of Aus¬ 
tria’s accession to the European Southern Observatory (ESO), 
we developed a comprehensive sky model 1 covering a wave¬ 
length range from 0.3 to 30 pm (Noll et al. 2012). It was origi- 

*http: //www. eso. org/observing/etc/skycalc/skycalc. 

htm 
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nally designed for the ESO Exposure Time Calculator 2 and in¬ 
corporates several components, such as airglow, scattered moon¬ 
light (Jones et al. 2013), zodiacal light (Leinert et al. 1998), scat¬ 
tered starlight, and the telescope emission modelled as a grey 
body. It also contains a spectral model of the Earth’s lower atmo¬ 
sphere in local thermal equilibrium calculated by means of the 
radiative transfer code LNFL/LBLRTM 3 (Clough et al. 2005). 
This code is used with the spectral line parameter database aer, 
which is based on HITRAN 4 (Rothman et al. 2009) and deliv¬ 
ered together with the code. Averaged atmospheric profiles for 
Cerro Paranal describing the chemical composition as a func¬ 
tion of height (combination of the ESO MeteoMonitor 5 , a stan¬ 
dard atmosphere 6 and the 3D Global Data Assimilation System 
(GDAS) model 7 * 9 ) are also used as input. The GDAS model is pro¬ 
vided by the National Oceanic and Atmospheric Administration 
(NOAA) 7 and contains time-dependent profiles of the tempera¬ 
ture, pressure, and humidity. 

The ESO In-Kind software also includes tools for removing 
the atmospheric signature in spectra 8-9 . The package molecfit 
is an alternative approach for performing the telluric absorp¬ 
tion correction by means of theoretically calculated transmis¬ 
sion spectra based on the atmospheric model. We use the pre¬ 
viously mentioned radiative transfer code and line database to 
derive synthetic transmission curves. The required atmospheric 
input profile is created in the same way as for the sky model, 
but for the time and location of the observations and not aver¬ 
aged. The algorithm and package functionalities are described 
in more detail in Smette et al. (2014) (hereafter Paper I). Here, 
we evaluate the quality of molecfit as a telluric absorption cor¬ 
rection tool for ESO Very Large Telescope (VLT) instruments. 
The focus lies on observations taken with the X-Shooter instru¬ 
ment (Vernet et al. 2011), an echelle spectrograph covering si¬ 
multaneously the wavelength regime from the U to the K band 
at medium resolution. We use two methods: (a) a statistical study 
of the quality of the correction by means of figures of merit, and 
(b) a comparison of the molecfit correction with that obtained 
using the traditional method based on observations of TSS. 

We first briefly describe the method of the telluric absorption 
correction incorporated in molecfit (Sect. 2). A description of 
the test data set is given in Sect. 3. In the following, we perform 
two different tests: In Sect. 4, we investigate the dependence of 
the telluric absorption correction quality on several parameters, 
i.e. the influence of the line transmission, observing conditions, 
spectral resolution, fitting ranges, input parameters, the atmo¬ 
spheric profiles, and the signal-to-noise ratio S /N. For this pur¬ 
pose, we introduce two figures of merit to estimate the quality of 
the telluric correction 7 0 ff and 7 res , describing the systematic off¬ 
sets and the remaining small-scale residuals of the corrections, 
respectively. These quantities are determined for a large data set 
of X-Shooter spectra, to which molecfit is applied with de¬ 
fault parameters optimised for this instrument. The results of this 
analysis allow the user to estimate the achievable quality for the 
telluric absorption correction in case an individual parameter set 
optimisation is not possible, e.g. due to the large number of spec¬ 
tra. In Sect. 5, we compare the classical method based on a TSS 
with molecfit as another test. We have selected four science 

2 http: //www. eso. org/observing/etc/ 

3 http://rtweb.aer.com/lblrtm_frame.html 

4 http://www.cfa.harvard.edu/hitran/ 

5 http://archive.eso.org/asm/ambient-server 

6 http: //www. atm. ox. ac. uk/RFM/atm/ 

7 http://www.ready.noaa.gov/gdas1.php 

s http: //www. eso. org/pipelines/skytools/ 

9 http://www.uibk.ac.at/eso/software/ 


Table 1 . Table of objects used for detailed tests. 


r 

object 

median counts 

slit 

Prog.-ID 


type 

[ADU] 

["X"] 


1 

Seyfert 2 

462 

1.5x11 

087.B-0614 

2 

GRB 

241 

0.9x11* 

091.C-0934 

3 

B[e] star 

7684 

0.4x11 

084.C-0952 

4 

E0 galaxy 

5236 

0.9x11 

384.B-1029 

5 

Carbon star 

50587 

0.9x11 

084.D-0117 

6 

PN 

6214 

0.4x11 

385.C-0720 


Notes. 

M Objects #1 and #2 are data with low S/N used for tests on the relia¬ 
bility of molecfit when applied to such data. #3 through #6 are used 
for a detailed comparison with the classical method. 

<bl observed with the K band blocking filter (Vernet et al. 2011) 

observations, for which both methods are optimised. In this way, 
individually optimised results can be compared for both meth¬ 
ods. The results of this comparison are demonstrated by focusing 
on the scientifically relevant spectral details. Section 6 provides 
a summary of our findings. 

2. Telluric correction with molecfit 

The correction with the molecfit package is performed in two 
steps: 

- Determination of the best-fit atmospheric and instrumental 
parameters: they are related to the total column densities of 
the input atmospheric profile and the instrumental param¬ 
eters (spectral line profile, wavelength calibration, contin¬ 
uum position) are optimised by means of a ^ 2 -Levenherg- 
Marquardt minimisation algorithm (see Paper I for a com¬ 
prehensive description of the underlying algorithms) to best 
reproduce the observed telluric spectrum in selected wave¬ 
length regions. By varying the scaling factor of the molec¬ 
ular profiles of the initial input atmospheric profile, the pro¬ 
gramme iteratively calculates transmission curves, which are 
fitted to the input science spectrum. To minimise the calcu¬ 
lation time, to optimise the continuum fit, and/or to avoid 
regions affected by stellar spectral features or instrument de¬ 
fects, the fitting is restricted to user-defined spectral ranges. 

- Correction of the telluric spectrum: the best-fit atmospheric 
and instrumental parameters are used to calculate the atmo¬ 
spheric transmission spectrum over the entire wavelength 
range of the scientific observation. The science spectrum is 
then divided by this transmission curve to produce telluric 
corrected data. 


3. The data set 

In order to evaluate the performance of molecfit, we have used 
archival data obtained with the X-Shooter instrument mounted 
at the ESO VLT (Vernet et al. 2011). This instrument covers 
the entire wavelength range from 0.3 to 2.5 pm in three spec¬ 
tral arms (UVB arm from the ultraviolet to the B band; VIS arm 
in the visual regime; NIR arm in the near-infrared regime), at 
medium resolution (R ~ 3300 to 18200, depending on the slit 
width) simultaneously. This broad wavelength regime gives the 
opportunity to study the correction of several absorption bands 
of different species simultaneously (e.g. HjO, O 2 , CO 2 ). 
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We have reduced the entire publically available ESO archive 
data from October 2009 to March 2013, leading to a compre¬ 
hensive data set taken under various observing conditions, since 
X-Shooter is frequently used. We used the ESO standard pipeline 
in version V2.0.0 under Reflex V2.3 on our cluster 10 . 

For studying the influence of the atmospheric conditions and 
instrumental set-ups on the quality of telluric absorption correc¬ 
tion (see Sect. 4), we have taken all ID spectra of TSS without 
flux calibration until March 2013. In total, there are 4218 NIR- 
arm and 3823 VIS-arm spectra. UVB-arm spectra were not con¬ 
sidered due to the lack of molecular absorption features * 11 . Due 
to occasional failures of the automatic X-Shooter pipeline, the 
number of obtained spectra is lower than the number of expo¬ 
sures in the archive. Since the obtained set of reduced data was 
large enough for our purpose, we did not attempt to re-run the 
pipeline manually (possibly using tuned reduction parameters) 
for the cases for which the automatic approach failed. 

Two observations of a y-ray burst and a Seyfert 2 galaxy (#1 
and 2 in Tables 1 and 2) were used to investigate the quality of 
the telluric absorption correction obtained with molecfit when 
applied to data with low S /N. Both observations show low count 
numbers (462 and 241 ADU, respectively) and were also taken 
at significant airmass, which increases the effect of atmospheric 
absorption. 

For the comparison with the classical method described in 
Sect. 5, we have selected science observations of four differ¬ 
ent astrophysical objects (#3 through 6 in Tables 1 and 2) in 
conjunction with their corresponding TSS observations manu¬ 
ally from our X-Shooter test data set in order to perform a com¬ 
parison between the telluric absorption correction achieved with 
molecfit and the classical method related to TSS. The science 
test data sets were selected to cover different astrophysical ob¬ 
jects and S/N traced by the counts in ADU (see Table 1) to also 
estimate the limits of the application of both methods. The TSS 
spectra were selected to coincide best with the airmass and the 
date of the corresponding science observation (see Table 2) and 
were usually reduced with the same flat field. As the majority of 
the telluric absorption features arise in the infrared regime, we 
restrict this investigation to NIR-arm data. 


4. Quality of telluric absorption correction 

In the following, we evaluate the quality of telluric absorp¬ 
tion correction with molecfit by means of a large sample of 
X-Shooter TSS spectra (see Sect. 3). We mainly focus on the 
NIR arm, where the telluric absorption correction is most crucial. 
We also complement the discussion with results from the VIS 
arm. The molecfit test set-up for the data set and the quality 
indicators used are discussed in Sect. 4.1. The results are shown 
in Sects. 4.2 to 4.4. The effect of changing the fitting ranges and 
molecfit input parameters is discussed for an example spec¬ 
trum in Sects. 4.5 and 4.6, respectively. The influence of differ¬ 
ences in the input water vapour profile is described in Sect. 4.7. 
Finally, we investigate the effect of low S /N data on the fitting 
quality in Sect. 4.8. 


Table 4. Wavelength ranges (vacuum) for fitting NIR- and VIS-arm 
X-Shooter TSS spectra 


Arm 

No. 

^min 

^max [/^m] 

Main molecule 

NIR 

1 

1.120 

1.130 

h 2 o 

NIR 

2 

1.470 

1.480 

h 2 o 

NIR 

3 

1.800 

1.810 

h 2 o 

NIR 

4 

2.060 

2.070 

co 2 

NIR 

5 

2.350 

2.360 

ch 4 

VIS 

1 

0.686 

0.694 

o 2 

VIS 

2 

0.759 

0.770 

o 2 

VIS 

3 

0.930 

0.945 

h 2 o 



Wavelength [/4m] 

Fig. 1. Reference X-Shooter NIR-arm model transmission spectrum 
binned in 1 nm stepsfor a mean amount of precipitable water vapour 
(PWV) of 3.1 mm and an airmass of 1. The bins marked by red sym¬ 
bols are classified as continuum. The transmission of these bins ranges 
from 0.95 to 1.00. The mean value is 0.99. The selected bins are used 
to derive the systematic offsets (residuals) in the telluric absorption cor¬ 
rected spectra. Bins that belong to four specific transmission ranges cor¬ 
responding to T = 0.9 (blue), 0.8 (cyan), 0.5 (magenta), and 0.2 (green) 
are indicated as well. The figure also shows the fitting ranges (yellow 
bars) that were used by molecfit (see also Table 4). 


4. 1. Test set-up and figures of merit 

The performance of molecfit was tested with a fixed input pa¬ 
rameter set. This approach is appropriate to estimate statistically 
the quality of the correction on a large set of data. Nevertheless, 
further quality improvements could be obtained by adjusting the 
fitting parameters for each individual spectrum. 

For the NIR arm, the applied set-up is shown in Table 3 (see 
also Paper I). For the wavelength range from 1 to 2.5 pm, the 
model-relevant molecules are O 2 , CO 2 , FDO, CH 4 , and CO. For 
the VIS arm, it is sufficient to consider O 2 and H 2 O. Only wa¬ 
ter vapour is fitted in spectra of both arms, since the concentra¬ 
tion variations and the impact on the X-Shooter data of the other 
species are expected to be small. The equatorial standard atmo¬ 
spheric profile that we use is already more than a decade old 
(prepared by J. J. Remedios 2001; see Seifahrt et al. 2010). As 
the global CO 2 concentration increases with time (World Meteo- 


10 12 x Intel Xeon X5650@2.67GHz/ 12GB RAM per node 

11 Note that the ozone absorption by the Huggins and Chappuis bands 
(see Paper I) is usually taken into account by the extinction correction 
(see e.g. Patat et al. 2011). 
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Table 2. X-Shooter NIR-arm data set used for the detailed tests (see also Table 1). 


Science observations Telluric standard stars 


r 

object 

airmass 

obs. date fsci 

star 

airmass 

obs. date Ass 

Afobs* [min] 

1 

PKS 1934-63 

1.69 

201 l-07-01T01:57:23 

— 

— 

— 

— 

2 

GRB 130606A 

1.715 

2013-06-07T04:09:16 

- 

- 

- 

- 

3 

V921-Sco 

1.109 

2010-03-10T09:01:02 

Hip084409 

1.058 

2010-03-10T09:42:39 

42 

4 

NGC5812 

1.06 

2010-03-13T08:57:03 

Hip073345 

1.128 

2010-03-13T08:48:11 

9 

5 

HE 1331-0247 

1.075 

2010-03-24T06:10:55 

Hip085195 

1.152 

2010-03-24T07:54:29 

104 

6 

IC1266 

1.222 

2010-07-05T05:54:38 

Hip085885 

1.248 

2010-07-05T06:07:41 

13 


Notes. 

Objects #1 and #2 are data with low S/N used for tests on the reliability of molecfit when applied to such data. #3 through #6 are used for a 
detailed comparison with the classical method. 

(A) approximate time between TSS and science target observations (Af obs =| t sa ~ ?tss I) 


Table 3. Molecfit parameter set-up for the telluric absorption correction evaluation of NIR-arm X-Shooter TSS spectra 


Parameter 3 

Value 

Short description 

WLGTOMICRON 

1(T 3 

factor to convert the input wavelength units into jum 

VAC_AIR 

air 

wavelengths in vacuum or air 

FTOL 

1(T 2 

relative^ 2 convergence criterion 

XTOL 

1(T 2 

relative parameter convergence criterion 

LIST_MOLEC 

O 2 CO 2 HoO CH 4 CO 

list of molecules to be included in the model 

FIT_MOLEC 

00 100 

flags for if each molecule is to be fit (1 =yes, 0=no) 

RELCOL 

1. 1.05 1. 1. 1. 

relative molecular column densities (1 = input profile) 

FIT_CONT 

1 

flag for polynomial fit of the continuum (1 =yes, 0=no) 

CONT_N 

1 

degree of polynomial for the continuum fit 

CONT_CONST 

10 4 

initial constant term for the continuum fit 

FIT_WLC 

1 

flag for refinement of wavelength solution (1 =yes, 0=no) 

WLC_N 

0 

degree of Chebyshev polynomial for refined wavelength 
solution 

WLC_CONST 

0 . 

initial constant term of the Chebyshev polynomial 

FIT_RE S_B OX 

1 

flag for resolution fit using a boxcar filter (1 =yes, 0=no) 

RELRES_BOX 

0.75 

initial value for FWHM of boxcar relative to slit width 
(> 0 and < 2) 

KERNMODE 

0 

kernel mode (0=independent Gaussian and Lorentzian, 

1 =Voigt profile) 

FIT_RE S_G AU S S 

1 

flag for resolution fit using a Gaussian filter (1 =yes, 0=no) 

RES_GAUSS 

1 . 

initial value for FWHM of Gaussian (in pixels) 

FIT_RES_LORENTZ 

0 

flag for resolution fit using a Lorentzian filter (1 =yes, 0=no) 

RES_LORENTZ 

0 . 

initial value for FWHM of Lorentzian (in pixels) 

KERNFAC 

3. 

kernel size in units of FWHM 

VARKERN 

1 

flag for selecting a constant (=0) or a variable kernel (=1) 

PIXSC 

0.20 

pixel scale in arcsec 


Notes. 

(a> Only parameters required for the fitting procedure and specific to the X-Shooter data are listed (see the molecfit User Manual for more details 


rological Organization 2012), the input CO 2 column was multi¬ 
plied by 1.05 to be representative of the X-Shooter archival data. 

A linear fit (cont_n =1) was performed to correct the con- 
tinua of the spectra in each of the fitting ranges (see below). The 
initial continuum factor cont_const was set to 10 4 to be in the 
order of the typical count level in ADU of X-Shooter data with¬ 
out flux calibration. For the wavelength grid correction, which 
is required to handle calibration uncertainties and an inaccurate 
centring of the target in the slit, only a constant shift was allowed 
(wlc_n = 0) (see Sect. 4.6 for a discussion). For the instrumental 


profile, a combination of a boxcar and Gaussian was assumed. 
A possible Lorentzian was not considered, since a study of the 
shape of the line profiles did not reveal significant Lorentzian 
wings. The width of the initial boxcar was chosen to be 75% 
of the slit width (relres_box), which should be close to the real 
value for the different slits (as confirmed by tests). An exception 
is the 5” slit, which is, however, rarely used for observations of 
science targets. For the Gaussian, a reasonable initial FWHM of 
1 pixel (res_gauss) and a kernel size of 3 FWHM (kernfac) were 
given. Since echelle spectra are fitted, the kernel FWHM was se- 
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1 1.5 2 2.5 

Wavelength [/xm] 


Fig. 2. Quality of telluric absorption correction for an example TSS 
spectrum taken with the NIR arm of the X-Shooter spectrograph. The 
star was observed with a 1.2” slit at an airmass of 1.32. The seeing 
was 0.76” and the PWV (as derived from the fitting) was 1.46 mm. Up¬ 
per panel: The telluric absorption corrected spectrum (red) is shown in 
comparison with the original spectrum (black). Middle and lower pan¬ 
els: Mean counts and standard deviation in ADU for a grid of 1 nm bins 
for the telluric absorption corrected example spectrum. The two subfig¬ 
ures also highlight the pixels for the continuum interpolation (red) and 
the different transmission ranges for the quality analysis of the telluric 
absorption correction. For more details, see Fig. 1. 


lected to be proportional to the wavelength (varkern =1). The 
profile-related input parameters refer to the central wavelengths 
of 1 .74 pm and 0.78 pm for the NIR and VIS arm, respectively. 
For the 214 spectra (5.1%) taken with a V-blocking filter (see 
Vernet et al. 2011), the corresponding wavelength is 1.55 pm. 

As shown by Table 4 and Fig. 1, the fitting of the atmospheric 
transmission model in the NIR arm was restricted to five nar¬ 
row 10 nm wide fitting ranges (or inclusion regions; cf. Paper I). 
They cover only about 3% of the entire wavelength range, which 
ensures that the fitting time is reasonable (typically 1 to 2 min) 
and the continuum fit with a low order polynomial is accurate 
enough. Moreover, only a relatively small fraction of the NIR- 
arm wavelength range is suitable for the fits, since a good fit re¬ 
quires a wide range of transmission values. The windows are suf¬ 
ficient to derive the amount of atmospheric water vapour (CCT 
and CH 4 are not fitted), the wavelength shift, and the instrumen¬ 
tal profile. Since the last two properties have to be determined for 
the entire spectrum. Ranges 4 and 5, which do not show signif¬ 
icant H 2 O absorption, are also important for a good coverage of 



Fig. 3. Indicators I 0 g for systematic offsets (upper panel) and 7 res for 
small-scale residuals (lower panel) of the telluric absorption correc¬ 
tion for single 1 nm bins of the X-Shooter NIR-arm spectrum shown 
in Fig. 2. Both quantities are given as a function of transmission. Bins 
belonging to the different transmission ranges used in the discussion are 
marked by different colours (see legend and Fig. 1). Small red symbols 
identify bins which are excluded from further analysis due to either very 
low transmission or very weak line strength. 

the full wavelength range (see also Sects. 4.5 and 4.6). Moreover, 
our optimised set of fitting ranges avoids absorption features of 
typical TSS. 

The VIS arm is much less affected by molecular absorption. 
The only prominent bands are the A and B molecular oxygen 
bands and the water vapour band at 0.94pm. Therefore, fitting 
ranges were only defined in these three bands as indicated by 
Table 4. The H 2 O-re I a ted range avoids the wavelengths of po¬ 
tentially strong Paschen lines, which are present in spectra of 
hot TSS. 

Running molecfit for the input parameters listed in Table 3 
and the fitting ranges shown in Table 4 results in the best-fit pa¬ 
rameters for each sample exposure (see Sect. 3). These are then 
used to provide a telluric absorption corrected spectrum for the 
full wavelength range (see Fig. 2). The quality of the fit in the 
pre-defined fitting ranges can be evaluated by considering the 
RMS of the residuals, which is provided by molecfit. However, 
the quality of the telluric absorption correction must be studied 
over the whole spectral range with respect to the quality of the 
corrected object spectrum and therefore requires a different anal¬ 
ysis. For this purpose, we have defined the figures of merit I 0 s 
and 7 res . The former measures the continuum-normalised differ¬ 
ence between the original and telluric absorption corrected spec¬ 
trum, relative to a locally-averaged telluric absorption strength. 
The latter traces the continuum-normalised standard deviation of 
the residuals of the correction, relative to a locally-averaged tel¬ 
luric absorption strength. Therefore, 7 0 g- and 7 res are indicators 
of large-scale systematic offsets in the telluric absorption cor¬ 
rected spectra and small-scale (or high-frequency) variations of 
the residuals, respectively. These quality indicators were calcu¬ 
lated in the following way: 

- First, each telluric absorption corrected NIR- and VIS-arm 

spectrum was divided into 1 nm and 0.5 nm wide bins, re¬ 
spectively (corresponding to about 17 and 25 pixels, i.e. a 

few resolution elements; cf. Sect. 4.4). 
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- The mean and the standard deviation were calculated for 
each of these bins (see Fig. 2). 

- The bins for which the average transmission model in Fig. 1 
shows no or only minor absorption were selected as contin¬ 
uum nodes. Even though this selection includes weak outer 
wings of some bands, this is not critical. Lines with a depth 
of about 1% can be well corrected (see Fig. 2), especially 
with respect to the much stronger lines for which the quality 
of the correction is evaluated. 

- The continuum bins were used to interpolate the object con¬ 
tinuum at the positions of bins with significant absorption. 

- The interpolated continuum intensity for each bin was then 
subtracted from the corresponding measured mean. This re¬ 
sults in an estimate of systematic offsets in the telluric ab¬ 
sorption corrected spectra. 

- To be independent of the absolute flux, the offsets and stan¬ 
dard deviations of each bin were divided by the interpolated 
continuum mean values. 

- To make the resulting values independent of the wavelength- 
and time-dependent transmission T, the relative offsets and 
standard deviations were divided by 1 — T, i.e. the average 
amount of absorption for each bin (see Fig. 3). 

- Five transmission ranges were defined for a detailed analysis 
of the telluric absorption correction. Apart from a wide range 
from 0.1 to 0.95, ranges centred around 0.9, 0.8, 0.5, and 0.2 
were defined (see Figs. 1 to 3). We did not consider T close 
to 1 and 0, since the related results are mainly noise driven 
and are therefore not suited to evaluate molecfit (see also 
Sects. 4.2 and 4.3). 

- The bins that belong to each of the five ranges were deter¬ 
mined by using the best-fit transmission spectra of the indi¬ 
vidual sample spectra. Thus, the bin assignment depends on 
the airmass, amount of atmospheric water vapour, and spec¬ 
tral resolution. 

- Finally, to obtain the figures of merit and to reduce the ef¬ 
fect of outliers, we took the median of the relative offset and 
standard deviation of the bins for each transmission range as 
shown in Fig. 3. 

The resulting quality indicators 7 0 ff and 7 res are used in the subse¬ 
quent analysis. If the transmission range is not specified explic¬ 
itly, the results for the wide range are given. Note that I Q s and 
7 res have to be multiplied by 1 - T to provide the systematic off¬ 
sets of the correction and the intra-bin standard deviations of the 
residuals relative to the object continuum. More details on the 
interpretation of our figures of merit can be found in Sect. 4.3. 

4.2. Outliers 

The indicator 7 res measures variations in the count level within 
each of the narrow bins of a telluric absorption corrected spec¬ 
trum. Primarily, this traces the small-scale quality of the telluric 
absorption correction. However, random noise, defects in the 
spectra, sky subtraction residuals, and spectral features of the 
observed object can also cause an increase of 7 res . This is demon¬ 
strated for random noise in Fig. 4, which shows a clear increase 
of the scatter for lower mean counts (calculated for the fitted 
pixels), i.e. decreasing S/N. In order to avoid difficulties in inter¬ 
preting the 7 res sample statistics, spectra with mean counts less 
than 10 4 ADU are excluded from further analysis. This threshold 
concerns 120 spectra of the NIR arm (i.e. 2.8% of the sample) 
and 36 spectra of the VIS arm (0.9%). Excluding these data does 
not mean that their corrected spectra have a bad quality. For ex¬ 
ample, the mean 7 0 ff for NIR-arm spectra with mean counts be¬ 
tween 10 3 and 10 4 ADU is 0.000 (cr = 0.053), i.e. there are no 
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Fig. 4. Small-scale residual indicator 7 rcs for 1 nm bins versus the mean 
model continuum counts in ADU for the five fitting ranges. The red 
dashed line shows the selection criterion for sufficiently high S/N for 
the detailed analysis. 

systematic continuum offsets on average. For a discussion of the 
quality of the telluric absorption correction for low-S/N spectra, 
see Sect. 4.8. 

The output files of the X-Shooter pipeline provide bad pixel 
masks, which can be used by molecfit to exclude critical pix¬ 
els from the fitting procedure. Sometimes in the NIR arm, it 
seems that more pixels were rejected by the pipeline than re¬ 
quired. In the case of a very small number of available pixels, 
the fit becomes unreliable. While the standard deviation might 
even decrease, systematic offsets are expected to become more 
significant. In addition, crucial continuum pixels for the interpo¬ 
lation of ranges with strong absorption bands could be missing, 
which makes the derivation of 7 0 ff less reliable. For this reason, 
we excluded 247 NIR-arm spectra (5.9%) with less than half the 
maximum number of pixels in the fitting ranges. 

So far, we have mainly rejected spectra where a proper cal¬ 
culation of the quality indicators 7)ir and 7 res could not be guaran¬ 
teed. However, for evaluating the quality of the telluric absorp¬ 
tion correction, it is important to know the fraction of obviously 
failed fits. For this purpose, we studied the best-fit FWHM of 
the instrumental profile (combined boxcar and Gaussian) and the 
best-fit shift of the wavelength grid. Interestingly, only a small 
number of NIR-arm spectra (no VIS-arm spectra) showed val¬ 
ues which were clearly separated from the general distribution. 
28 best-fit model spectra with a FWHM above 10 pixels (or be¬ 
low 1.5 pixels) or wavelength shifts of more than 2.5 pixels rel¬ 
ative to the sample mean could be identified as clear outliers. 14 
of these 28 spectra were already rejected by the critera described 
above. Molecfit appears to show a very robust performance, at 
least for X-Shooter TSS spectra. 

For further analysis, we excluded all the discussed spectra 
with suspicious fits. This results in subsamples of 3837 NIR-arm 
(91% of the full sample) and 3787 VIS-arm spectra (99%). 

4.3. Influence of line transmission and observing conditions 

For the filtered NIR-arm sample (see Sect. 4.2), Table 5 shows 
the mean values and standard deviations of 7 0 fj and 7 res for the 
five transmission ranges listed. In addition, the individual val¬ 
ues for the wide transmission range from 0.1 to 0.95 are plotted 
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Table 5. Sample averages and standard deviations for the transmission- 
dependent indicators I 0 g and 7 res of the quality of the telluric absorption 
correction of NIR-arm X-Shooter spectra 


Ref. T T range 


4>ff 


mean 


*res 

mean cr 


0.9 

0.8 

0.5 

0.2 


0.10-0.95 
0.88-0.92 
0.75-0.85 
0.45 - 0.55 
0.15-0.25 


-0.009 

-0.008 

-0.024 

-0.004 

+0.035 


0.032 

0.044 

0.046 

0.038 

0.063 


0.196 

0.209 

0.184 

0.150 

0.237 


0.063 

0.071 

0.068 

0.059 

0.136 


0.4 - 


0.3 - 


0.2 - 


0.1 
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- 0.1 
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Fig. 5. Indicators I oS for systematic offsets and 7 res for small-scale resid¬ 
uals of the telluric absorption correction for 1 nm bins. Only data points 
of the filtered subsample are shown. 




Fig. 6. Fraction of NIR-arm bins with standard deviations of the resid¬ 
uals of the telluric absorption correction greater than or equal to 5% of 
the object continuum as function of the best-fit PWV in mm. The red 
regression line has a slope of 0.15 per dex. 


Fig. 7. Best-fit PWV obtained by molecfit in mm as function of the 
observing date in years. 


in Fig. 5. The latter does not indicate significant features in the 
distribution of the data points. 7 0 ff clusters around a value of 0, 
which means that the telluric absorption correction does not ap¬ 
pear to be affected by systematic offsets. The scatter is only about 
3% of the line strengths. The mean 7 res is about 0.20 with a scat¬ 
ter of 0.06, i.e. the relative standard deviation of the residuals 
of the corrected telluric absorption is about 20%. This translates 
into typical errors of 2% and 10% relative to the continuum for 
spectral ranges with T = 0.9 and 0.5, respectively. The mean 7 0 ff 
and 7 res values for the X-Shooter VIS arm -0.002 (cr = 0.048) 
and 0.184 (cr = 0.059) are very similar to the NIR-arm results. 
This implies that the telluric absorption correction is of good 
quality in both X-Shooter arms. The results are consistent with a 
typical correction accuracy of 2% of the continuum or better for 
wavelength ranges with unsaturated telluric lines as reported in 
Paper I based on data from different instruments. Note that indi¬ 
vidual molecular lines are not resolved in the X-Shooter spectra, 
which lowers and smoothes the measured telluric absorption. 

Table 5 lists / 0 ii and 7 res depending on the four narrow trans¬ 
mission ranges centred at T — 0.9,0.8,0.5, and 0.2. The distribu¬ 
tions agree quite well with the results for the wide transmission 
range. Consequently, the mixing of transmissions is not crucial 
for the resulting quality indicators, at least if T very close to 1 
and 0 are not considered (as for our wide T range). In the case 
of very high T, the figures of merit are no longer reliable with 
respect to the quality of the telluric absorption correction, since 
random noise, systematic errors in the reduced spectra, and fea¬ 
tures of the observed object can have a strong effect due to the 
normalisation by 1 - T (see Fig. 3). The correction quality is 
probably comparable with the results for intermediate T, which 
are crucial for the model fit for all T. In the case of very low T, 
the relatively low S/N, possible zeropoint errors, and the strong 
variation in T over narrow wavelength ranges can cause high 7 0 ff 
and 7 res . The correction of spectral ranges with T close to 0 is 
difficult (see Fig. 2). However, this is usually not an issue, since 
the information from the science target also tends to be very lim¬ 
ited. The 7 res values in Table 5 illustrate the described effects. 
The minimum of 0.15 is obtained for intermediate transmissions 
(T = 0.5), whereas the values for T = 0.9 and 0.2 are above 0.2. 

The rough proportionality of the telluric absorption correc¬ 
tion errors relative to the continuum and 1 - T for a wide range 
of T implies that the overall correction quality of a spectrum 
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is correlated with properties which affect T, i.e. the airmass of 
the observation and the column density of the molecular species 
concerned. This can be understood by considering that the trans¬ 
mission T is related to the optical depth along the line of sight r 
for each wavelength A by 

T(A) = e~ rU> , (1) 

where r can be approximated by the product of the optical depth 
at zenith r q and the airmass X: 

t(A)*t 0 {A)X. (2) 

This works best if the geometrical distributions of the given 
molecule and air are similar. Finally, the optical depth ro for 
molecular absorption by a single species at wavelength A can 
be calculated by 

/-*o o 

t 0 (A) = cr abs (A) ( n(h) d h, (3) 

Jho 

where cr a b s is the wavelength-dependent absorption cross sec¬ 
tion and the integral corresponds to the column density of the 
molecule, which is derived from the density n at heights h above 
the observer at /to- Consequently, telluric absorption correction 
is most difficult if a target is observed at a large zenith distance 
and with a high atmospheric water vapour content. The amount 
of water vapour is critical, since most prominent bands in the 
X-Shooter wavelength range are caused by this molecule and 
the concentration and distribution is highly variable in time and 
space. 

The effect of water vapour on the quality of the telluric ab¬ 
sorption correction is shown in Fig. 6, which displays the frac¬ 
tion of 1 nm NIR-arm bins with residuals greater than or equal 
to 5% of the corrected object continuum as function of the 
amount of precipitable water vapour (PWV) in mm as derived by 
molecfit. For our X-Shooter NIR-arm TSS sample, the fitted 
PWV range from 0.2 to 18.2 mm with a mean value of 3.1 mm. 
The median is 2.2 mm. These values are in good agreement with 
other measurements of the PWV, which is routinely monitored at 
Cerro Paranal by a stand-alone microwave radiometer in support 
of science observations (see Kerber et al. 2014, and Sect. 4.7). 
There is a clear, nearly linear increase of the number of bins with 
significant residuals with increasing PWV in logarithmic units. 
A regression analysis indicates that the fraction /o->o ,05 grows 
from about 0.20 at 1 mm to about 0.35 at 10 mm. For compari¬ 
son, the fraction of bins with a transmission lower than 0.95 of 
the mean spectrum plotted in Fig. 1 is about 0.5. The effect of 
the airmass on /o->o.o 5 is smaller than for the PWV, since the air¬ 
mass only varies by a factor of about 2. In the VIS-arm range, the 
regression line exhibits an increase of /o-> 0.05 from about 0.04 at 
1 mm to 0.07 at 10 mm. Since strong (water vapour) lines are rare 
in this wavelength regime, the fractions are distinctly smaller 
than in the NIR-arm range. 

The quality of the correction of water vapour bands may 
roughly depend on the time of the year. Figure 7 indicates a 
strong seasonal dependence of the atmospheric water vapour 
abundance. In winter, the PWV is relatively low (mean of 
1.9 mm for meteorological winter), whereas the highest amounts 
and a large scatter are found in summer (mean of 5.2 mm). The 
strength of molecular oxygen and carbon dioxide bands can be 
considered as nearly stable except for the long-term increase of 
the CO 2 concentration (see Sect. 4.1). 


Table 6. Slit-dependent quality of the telluric absorption correction of 
NIR-arm X-Shooter spectra 


Slit 

[”] 

N 

FWHM 3 
[pixels] 
mean cr 

4)if 

mean 

cr 

^res 

mean cr 

0.4 

437 

2.49 

0.20 

+0.008 

0.059 

0.239 

0.069 

0.6 

711 

3.07 

0.31 

-0.002 

0.020 

0.199 

0.047 

0.9 

1927 

3.95 

0.74 

-0.012 

0.027 

0.193 

0.064 

1.2 

566 

4.44 

1.01 

-0.015 

0.024 

0.179 

0.056 

1.5 

148 

5.03 

1.62 

-0.020 

0.020 

0.158 

0.059 

5.0 

48 

5.41 

1.57 

-0.057 

0.025 

0.156 

0.066 


Notes. 

(a) S ince the FWHM in pixels depends on the wavelength for a nearly 
constant spectral resolution, the FWHM is given for the centre of the 
full NIR-arm wavelength range, i.e. 1.74pm (see also Sect. 4.1). For 
spectra taken with a X-blocking filter, which only extend up to 2.1 pm, 
the FWHM was corrected to be also representative of 1.74 pm. 



Fig. 8. Small-scale residual indicator 7 res for 1 nm bins versus the 
FWHM of the line profile in pixels. As indicated by the legend, the 
different colours correspond to different slit widths. 

4.4. Influence of resolution 

Since the echelle gratings of the X-Shooter spectrographs are 
fixed, the slit width is the only instrumental parameter that af¬ 
fects the line-spread functions of the resulting spectra. For the 
NIR and VIS arms, seven different widths from 0.4” to 5.0” 
can be selected as displayed in Table 6 12 . Apart from the slit 
selection, the positioning accuracy of the target in the slit and its 
change with time contribute to the resulting line-spread function. 
Finally, the FWHM of a line is influenced by the seeing at the 
time of the observation of the point-like standard star, especially 
if the slit is larger than the light profile of the target. For this rea¬ 
son, Table 6 shows an increase of the FWHM (as derived from 
the combined best-fit boxcar and Gaussian kernels, see Sect. 4.1) 
as well as its scatter with increasing slit width for the NIR arm. 

The slit-dependent results for the quality indicators I 0 ff and 
/ res for the wide transmission range are also provided by Ta¬ 
ble 6. The systematic offsets appear to indicate a weak trend 
from slight overcorrection for the 0.4” slit to moderate under- 

12 In the case of the VIS arm, the 0.6” slit is replaced with 0.7”. 
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Table 7. Influence of fitting ranges on the telluric absorption correction 
of the NIR-arm spectrum displayed in Fig. 2 


Run 

Ranges 1 * 

Rel. 

RMS 

FWHM b 

[pixels] 

PWV 

[mm] 

foff 

^res 

1 

12345 

0.054 

3.76 

1.46 

-0.008 

0.235 

2 

1- 

0.030 

3.19 

1.68 

+0.085 

0.349 

3 

-2- 

0.022 

2.60 

1.32 

-0.063 

0.246 

4 

— 3 — 

0.028 

3.08 

1.40 

-0.041 

0.222 

5 

123 — 

0.066 

3.79 

1.48 

-0.004 

0.238 

6 

1-5 

0.035 

3.29 

1.56 

+0.039 

0.315 

7 

i 

i 

CN 

1 

0.034 

3.28 

1.36 

-0.036 

0.235 


Notes. 

M p or t }j e wavelength limits of the different ranges with the indicated 
numbers, see Table 4. 

(b) The FWHM is given for the centre of the full NIR-arm wavelength 
range, i.e. 1.74 //m. 

correction for the 5.0” slit. However, except for the value for 
the rarely used 5.0” slit, the mean offsets can be considered as 
negligible. Nevertheless, the slit width seems to contribute to a 
broadening of the 7 0 ff distribution of the entire data set (see Fig. 5 
and Table 5). 

As indicated by Fig. 8 and Table 6, the intra-bin variations of 
the residuals of the telluric absorption correction increase with 
decreasing slit width or FWHM. The mean 7 res values range 
from 0.156 for the 5.0” slit to 0.239 for the 0.4” slit. For the 
VIS arm, the corresponding values are 0.153 and 0.219. This 
clear dependence broadens the distribution of 7 res for the entire 
data set (see Fig. 5 and Table 5). At first, the trend can be ex¬ 
plained by the expected steepness of the line profiles if a line 
comprises only a few pixels (about 2.5 pixels for the 0.4” slit of 
the NIR arm). In this case, small discrepancies between the mod¬ 
elled and the true profile can cause significant residuals. On the 
other hand, the scatter in the NIR/VIS arm is calculated for bins 
of a width of 1 nm/0.5nm (about 17/25 pixels). If the FWHM 
is low, more (probably uncorrelated) resolution elements fit into 
the bin range. This effect could also augment 7 res . 

4.5. Influence of fitting ranges 

So far, the discussion has been based on a fixed set of fitting 
ranges (see Table 4 and Sect. 4.1). In the following, we focus on 
changes in the quality of the telluric absorption correction when 
these ranges are changed. For this purpose, we tested the NIR- 
arm example spectrum shown in Fig. 2, which is characterised 
by an airmass of 1.32 and a best-fit PWV of 1.46 mm. We ran 
molecfit for different subsets of the five standard NIR-arm fit¬ 
ting ranges, which had to include at least one range dominated 
by water vapour lines. 

Table 7 shows the results of the seven test runs we performed. 
The table indicates the selected fitting ranges (see Table 4 for 
the numbers), the RMS of the fit residuals relative to the mean 
counts, the best-fit FWHM, the best-fit PWV, and the two qual¬ 
ity indicators 7 0 ff and 7 res (see Sect. 4.1). The table entries for the 
different parameters show a clear difference in the quality of the 
fit and the telluric absorption correction depending on the fitting 
ranges considered. As expected, the RMS of the fit is reduced 
if the number of the fitting ranges is decreased. The resulting 
FWHM range from 69% to 101% of the value of the standard 
run. For the PWV, we obtained values from 90% to 115%. The 


Table 8. Influence of changing fitting ranges in a molecular band on 
the telluric absorption correction of the NIR-arm spectrum displayed in 
Fig. 2 


Run 

Range 

[pm] 

Rel. 

RMS 

FWHM a 

[pixels] 

PWV 

[mm] 


^res 

2 b 

1.12-1.13 

0.030 

3.19 

1.68 

+0.085 

0.349 

2a 

1.13-1.14 

0.033 

3.21 

1.72 

+0.101 

0.399 

2b 

1.14-1.15 

0.039 

3.03 

1.42 

+0.061 

0.502 

2c 

1.12-1.15 

0.041 

3.25 

1.56 

+0.100 

0.494 

3 b 

1.47-1.48 

0.022 

2.60 

1.32 

-0.063 

0.246 

3a 

1.46-1.47 

0.021 

2.78 

1.37 

-0.044 

0.237 

3b 

1.45-1.46 

0.023 

2.97 

1.35 

-0.067 

0.256 

3c 

1.44-1.45 

0.023 

2.97 

1.35 

-0.067 

0.256 

3d 

1.44-1.48 

0.036 

2.82 

1.37 

-0.045 

0.242 


Notes. 

T he FWHM is given for the centre of the full NIR-arm wavelength 
range, i.e. 1.74 pm. 

same runs as in Table 7 



1 1.6 2 2.5 

Wavelength [/zm] 


Fig. 9. Mean counts in ADU and standard deviation relative to mean 
counts for a grid of 1 nm bins of the telluric absorption corrected exam¬ 
ple spectrum shown in Fig. 2 for three different sets of fitting ranges. 
The black spectrum equals the one in Fig. 2 and corresponds to the 
standard set-up of windows described in Table 4. The red and the green 
spectra were calculated by only using a single fitting range (see legend 
and Table 7). 


largest deviations are found for runs that were based on only one 
fitting range. For the quality of the telluric absorption correc¬ 
tion as measured by 7 0 ff and 7 res , there is a similar trend. How¬ 
ever, the quality of the correction also strongly depends on the 
ranges involved. While the fit only depending on Range 1 (1.12 
to 1.13 pm) is by far the worst (Run 2), the result for Run 4, 
which is only based on Range 3(1.80 to 1.81 pm), is remarkably 
good. This is also illustrated by Fig. 9, which shows the mean 
values and relative standard deviations for 1 nm bins of the re¬ 
sulting spectra of Run 2 and 4 in comparison with the standard 
Run 1. Run 2 only led to a good telluric absorption correction 
in the fitted range, whereas Run 4 shows a reasonable correction 
over the entire wavelength range. 
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Table 9. Influence of wavelength grid correction by a Chebyshev poly¬ 
nomial of order wlc_n on the telluric absorption correction of the spec¬ 
trum displayed in Fig. 2 


WLC_N 

Rel. 

RMS 

FWHM 

[pixels] 

PWV 

[mm] 

foff 

^res 

0 

0.054 

3.76 

1.46 

-0.008 

0.235 

l 

0.126 

7.25 

1.51 

-0.046 

0.456 

3 

0.121 

7.05 

1.51 

-0.053 

0.459 

5 

0.120 

6.98 

1.51 

-0.050 

0.453 


Table 10. Influence of wavelength grid correction by a Chebyshev 
polynomial of order wlc_n on the telluric absorption correction of the 
spectrum displayed in Fig. 2 for the fixed best-fit line profile (rel- 
res_box = 0.560 and res_gauss = 1.671) as derived from the standard 
test run 


wlc_n = 013 


_ J, _ ,*^4^ 

_J_1_1_1_ 

L Ji 

_i_l_i_i_i_ 

IfejM _i_ ilwJtJil: 

_i_1_i_i_i_i_ 


j_i_i_i_i_i_i_i_i_i_i_i_i_i_i_ 

1 1.5 2 2.5 

Wavelength [/zm] 


WLC_N 

Rel. 

RMS 

FWHM 

[pixels] 

PWV 

[mm] 

foff 

^res 

0 

0.054 

3.75 

1.47 

-0.007 

0.235 

l 

0.031 

3.75 

1.44 

-0.028 

0.195 

3 

0.040 

3.75 

1.39 

-0.042 

0.262 

5 

0.031 

3.75 

1.42 

-0.021 

0.343 


Since the results for the molecfit runs with single fitting 
ranges differ significantly, we performed the model fitting and 
telluric absorption correction for single ranges that cover telluric 
lines in the same bands such as Runs 2 and 3, but with differ¬ 
ent wavelength limits. Only ranges where lines of intermediate 
strength dominate were selected (see Fig. 1). For ranges within 
the same HiO band. Table 8 reveals similar values for the listed 
parameters. This suggests that changing centre and width of a fit¬ 
ting range in a band (in a reasonable way) has less of an impact 
than changing the band. 

These results imply that modifying the fitting range within 
the HiO band at 1.13 pm does not significantly improve the qual¬ 
ity of the fit. The PWV, 7 0 ff, and 7 res values remain unsatisfying 
if a fitting range within this band is not combined with ranges 
in other molecular bands. Since the line depths of the different 
ranges are comparable, this does not seem to explain the dis¬ 
crepancies. Differences in the best-fit line shapes and wavelength 
shifts (up to 1 pixel) for the different molecfit runs could imply 
that the issue is linked to the structure of the X-Shooter compos¬ 
ite echelle spectra consisting of many orders. At least, line pro¬ 
files in the 1.13 pm band cover less pixels than those in bands at 
longer wavelengths (see Sect. 4.4). In view of the uncertainties 
in the line profile and wavelength calibration, a good telluric ab¬ 
sorption correction over the entire wavelength range (see Fig. 9) 
requires that all critical molecular absorption bands are probed 
by fitting ranges. Therefore, our standard set of fitting windows 
(see Table 4) is well defined, even though only low order cor¬ 
rections of the different kinds of systematic deviations from the 
atmospheric transmission model are possible (see Sect. 4.6). The 
best fit is always the result of a compromise, as indicated by the 
smaller residuals for individual fitting ranges in the correspond¬ 
ing wavelength regimes (see Fig. 9). 


Fig. 10. Mean counts in ADU and standard deviation relative to mean 
counts for a grid of 1 nm bins of the telluric absorption corrected ex¬ 
ample spectrum shown in Fig. 2 for different orders of the Chebyshev 
polynomial for the wavelength grid correction and fixed line profile (see 
legend and Table 10). 


4.6. Influence of input parameters 

Next, we investigate the influence of the input fit parameters on 
the quality of the telluric absorption correction. In this respect, 
our input parameter set (see Table 3) appears to be a reason¬ 
able choice (see discussion in Sect. 4.1). However, there could 
be a critical restriction of the maximum order of the Chebyshev 
polynomial for the correction of the wavelength grid. The selec¬ 
tion wlc_n = 0 only allows a constant shift of the wavelength 
grid. As discussed in Sect. 4.5, this is most likely not sufficient 
to achieve good fits in all the different fitting ranges at the same 
time. For this reason, we studied the effect of the degree of the 
Chebyshev polynomial on the quality of the fit. Table 9 shows 
the results of our investigation of the standard NIR-arm exam¬ 
ple (see Fig. 2) for four different degrees of the polynomial. The 
result columns are the same as in Table 7. The values for the rel¬ 
ative RMS, FWHM, 7 0 ff, and 7 res clearly imply that the fits of the 
runs with wlc_n =1,3, and 5 failed. The doubling of the FWHM 
for these runs indicates an increase of the degrees of freedom by 
the additional coefficients of the Chebyshev polynomial caused 
degeneracies, which led to an erroneous fit of the instrumental 
profile. In other words, the fitting algorithm was not able to find 
the global^ 2 minimum. 

To make the wavelength correction more robust, we per¬ 
formed a second series of runs with reduced degrees of free¬ 
dom. For this purpose, we fixed the properties of the line pro¬ 
file. We took the best-fit results of the standard run and set rel- 
res_box = 0.560, res_gauss = 1.671, and the corresponding fit 
flags to 0 (cf. Table 3). The results for the four different de¬ 
grees of the Chebyshev polynomial are listed in Table 10. For the 
higher wlc_n, the fits in the five fitting ranges are now better than 
for the standard run, as the relative RMS indicate. The PWV val¬ 
ues and the related 7 0 ff for systematic offsets are relatively stable. 
The small-scale residuals indicator 7 res is the lowest (0.195) for 
a linear wavelength correction function, i.e. wlc_n = 1. Higher 
order corrections indicate worse 7 res (0.343 for wlc_n = 5). They 
tend to deteroriate the telluric absorption correction, despite the 
fixed line profile. This is also demonstrated by Fig. 10, which 
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Best-fit scaling factor for input HsO profile 

Fig. 11. PWV in mm versus the best-fit scaling factor for the input water 
vapour profile (output value for relcol parameter; see Table 3). 

shows the mean values and relative standard deviations for 1 nm 
bins of the telluric absorption corrected spectra for the different 
runs. For higher degrees of the polynomial, the quality of the 
correction seems to be highly wavelength dependent. 

These findings suggest that very high wlc_n are risky be¬ 
cause of fit degeneracies. The situation could improve if there 
were more and/or broader fitting ranges. However, for the wave¬ 
length range covered by X-Shooter, this approach is not feasi¬ 
ble due to the small fraction of wavelengths with suitably strong 
absorption lines. Irrespective of these issues, the example has 
shown that a combination of two runs (where the second run 
benefits from the results of the first run) could significantly im¬ 
prove the quality of the telluric absorption correction. 

For a successful^- 2 minimisation, the number of free param¬ 
eters should not be too high. Apart from the wavelength grid cor¬ 
rection, the line shape parameters are prone to % 2 degeneracies. 
Line blends by low spectral resolution and only small numbers 
of telluric absorption lines by very narrow fitting ranges can be 
critical. Fortunately, the X-Shooter spectra do not seem to re¬ 
quire modelling of possible broad line wings by a Lorentzian 
kernel component (see Table 3), which makes the fits more ro¬ 
bust. 

4.7. Influence of the input atmospheric profile 

Water vapour is an abundant and very variable component of the 
Earth’s atmosphere (see Fig. 7). Most telluric absorption in the 
X-Shooter NIR-arm range is caused by this molecule. Therefore, 
the quality of the telluric absorption correction strongly depends 
on a good fit of the water vapour column density. 

The PWV fit corresponds to a scaling of the input water 
vapour profile. The parameter relcol describes the relative scal¬ 
ing with respect to the input profile. Fig. 11 shows the final PWV 
value from the best fit versus the best-fit relcol (see Table 3) 
for the NIR-arm data set selected in Sect. 4.2. The mean factor 
is 0.87 with a scatter of 0.26. This is relatively close to 1, i.e. the 
case that the PWV of the input profile is the best-fit one. How¬ 
ever, for low PWV, the merged input profiles tend to have too 
much water vapour. For PWV below the median value of 2.2, 
the mean scaling factor is 0.73, whereas for PWV above the me¬ 
dian, a mean factor of 1.01 is obtained. The standard deviation is 
similar in both cases (0.22 versus 0.23). In view of the significant 


scatter and the systematic offsets at low PWV, a reliable scaling 
of the input profiles is indispensable. 

To more accurately investigate the effect of the initial atmo¬ 
spheric profiles, we used data of a radiometer, which was in¬ 
stalled on Cerro Paranal in October 2011 for water vapour mon¬ 
itoring purposes (Kerber et al. 2012a,b). It is a Low Humid¬ 
ity And Temperature PROfiling microwave radiometer (LHAT- 
PRO), manufactured by Radiometer Physics GmbH (RPG 13 ). 
The instrument uses several channels across the strong water 
vapour emission line at 183 GHz, necessary for measuring the 
low levels of PWV that are common on Cerro Paranal. Details 
of the radiometer are described in Rose et al. (2005). This ra¬ 
diometer provides continuous direct on-site measurements of the 
temperature and water vapour content. It also calculates the pres¬ 
sure profile up to a height of about 12 km above Cerro Paranal, 
and the integrated water vapour (IWV), identical to the PWV. 
We created an additional set of initial atmospheric profiles for 
molecfit in the same way as the default combination of a stan¬ 
dard atmosphere, GDAS model, and ESO MeteoMonitor de¬ 
scribed in Paper I, but replaced the GDAS by the radiometer 
profiles and skipped the MeteoMonitor data since the latter pro¬ 
files already contain this information. 

From our X-Shooter data set, we selected 549 telluric stan¬ 
dard star observations obtained in Jan, Feb, and Aug-Dec 2012, 
which correspond to the period of the radiometer data provided 
to us by ESO. Every spectrum was fitted with molecfit in¬ 
corporating both, the GDAS/MeteoMonitor and the LHATPRO 
based set, respectively. In both cases, we used the closest avail¬ 
able profiles. We finally applied a telluric absorption correction 
based on both methods. 

The telluric corrected spectra differ usually by only a few 
per cent, with larger deviations in the ranges affected by strong 
atmospheric absorption (see Fig. 12 for an example). 

For a closer look, we also compared the resulting relcol 
values and the water vapour content values, PWV and IWV 
respectively, for the whole data set. Figure 13 gives the com¬ 
parison of the relcol parameters between the GDAS and the 
LHATPRO based fits. As expected, the LHATPRO based scal¬ 
ing parameters are closer to unity (median value = 0.95) than 
the relcol parameter derived with the help of the GDAS model 
(median value = 0.92). Also the relcol scatter <x H = 0.13 for 
the LHATPRO method is significantly lower than the GDAS 
based scatter (ctg = 0.25). This means that the atmospheric pro¬ 
file based on the radiometer data requires less scaling than the 
modelled one. This is expected, since the LHATPRO profiles 
are direct on-site measurements providing more accurate esti¬ 
mates of the actual atmospheric conditions than the combined 
GDAS/MeteoMonitor model. 

Although the radiometer based initial profiles lead to less 
scatter in the relcol parameter, the quality of the final telluric 
absorption correction is comparable within a few per cent. This 
means that due to the adaption with the scaling parameter, an 
inappropriate initial atmospheric profile also leads to a good tel¬ 
luric absorption correction. We therefore conclude that the un¬ 
derlying fitting algorithm incorporated in molecfit is highly 
efficient. However, it appears that the derived PWV value is over¬ 
estimating the real water vapour content in the case of very dry 
observing conditions. This indicates inadequate input profiles. 


13 http://www.radiometer-physics.de/ 
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Fig. 12. Relative difference of the telluric absorption corrected spectra 
of the TSS Hip039634 (IWV= 1.5 mm), one corrected with a LHAT- 
PRO, and one with a GDAS based profile (in per cent), ctsn = counts 
of the LHATPRO based telluric absorption corrected spectrum; cts G = 
GDAS counterpart. 
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Fig. 13. Comparison of the relative water vapour column scaling factors 
relcol derived with the standard GDAS/MeteoMonitor model and the 
LHATPRO measurements for 549 TSS spectra. 


4.8. Influence of the S/N 

In this section, we investigate the performance of molecfit with 
respect to low signal data. For this purpose, we used a spectrum 
of the y-ray burst GRB 130606A (Xu et al. 2013; Ukwatta et al. 
2013; Castro-Tirado et al. 2013; de Ugarte Postigo et al. 2014) 
at redshift z ~ 5.913 (Hartoog et al. 2014) taken in lune 2013 
(Prog.-ID: 091.C-0934; PI.: Kaper). Fig. 14 shows the uncor¬ 
rected (upper panel), the transmission (middle panel), and the 
corrected spectrum (lower panel). Although the object flux has a 
low mean count level of 241 ADU, molecfit was able to reli¬ 
ably correct the telluric absorption in wide parts. 

Another low S /N example is the spectrum of PKS1934-63, a 
Seyfert 2 galaxy, taken in luly 2011 (Prog.-ID: 087.B-0614; PI.: 
Holt). These data contain a comparably low median count level 
of only 462 ADU (c.f. Table 1). Fig. 15 shows as an example 
the three wavelength ranges 1.1 to 1.3 pm (upper panel), 1.3 to 
1.5/mi (middle panel) and 1.9 to 2.1 pm (lower panel), respec- 



wavelength [pm] 


Fig. 14. X-Shooter NIR-arm spectrum of y-ray burst GRB 130606A at 
z = 5.913. The graphs show the wavelength range from 1.15 to 1.16/im, 
which is affected by both telluric and some intrinsic absorption lines 
arising at a redshift z = 3.4515 (Hartoog et al. 2014). Upper panel: the 
uncorrected spectrum. Middle panel: transmission spectrum as calcu¬ 
lated by molecfit. Lower panel: corrected spectrum. 


tively. The first two ranges covers the strong water vapour ab¬ 
sorption bands between 1.1 and 1.17 pm and 1.34 and 1.52 pm, 
respectively. The correction for minor absorption regions is rea¬ 
sonable. The prominent CCL features between 2.0 to 2.08 pm 
plotted in the lower panel are corrected well. 

However, strong absorption bands cause major problems 
when correcting low ADU data. Fig. 15 also shows the strong 
absorption bands between 1.34 to 1.52 pm (panel (a)) and 1.8 to 
1.94 pm (panel (b)) arising from water vapour and carbon diox¬ 
ide, respectively. In these regions, the correction leads to large 
uncertainties, since the incorporated division and the low signal 
level increase the noise significantly. We therefore conclude that 
the telluric absorption correction with molecfit can be critical 
in such cases. For very low S /N data it might be even impos¬ 
sible to apply molecfit. In this case, the user might consider 
observing TSS to derive the correction function. 


5. Comparison with the classical method 

5.1. Method 

The classical way of the telluric absorption correction is done 
with the help of TSS, which are used to derive the transmission 
of the Earth’s atmosphere at the time of the observation. Since 
TSS are stars with few or well known spectral features, they can 
be used to obtain an atmospheric transmission after subtracting 
their continuum and subsequent normalisation. We used the fol¬ 
lowing approach: We fitted a cubic spline to base points selected 
on positions with or close to transmission T — 1 in the TSS 
spectrum in order to determine the continuum. This spline fit is 
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Fig. 15. X-Shooter NIR-arm spectrum of the Seyfert 2 galaxy 
PKS1934-63 (z ~ 0.183). The upper panels of (a), (b), and "of (c) 
show the uncorrected and the molecfit corrected spectrum in different 
wavelength ranges. This observation has a low mean ADU level of 462 
counts. The displayed transmission curve (lower panels) is based on a 
fit in the standard fitting regions (yellow areas). 


then used to normalise the TSS spectrum resulting in a transmis¬ 
sion spectrum (see Fig. 16). We corrected the science spectrum 
with this transmission using the IRAF task telluric 14 . This 
task also performs a wavelength shift and a scaling of the input 
spectrum to achieve the telluric absorption correction. 

For the comparison of the quality of the telluric absorption 
correction between classical TSS and molecfit methods, we 
used X-Shooter NIR-arm spectra of four different scientific ob¬ 
jects, a B(e) star, an E0 galaxy, a Carbon star, and a planetary 
nebula (PN), in conjunction with their corresponding telluric 
standard stars (see Tables 1 and 2). We divided the wavelength 
range covered by the NIR arm into 16 pieces: eight regions with 
major atmospheric absorption features (labelled with the num¬ 
bers #1 through #8) and another eight regions with minor ab¬ 
sorption (labelled with the letters ’a’ through ’h\ see Fig. 16 and 
Table 11). To optimise the telluric absorption correction, an in¬ 
dividual set of fitting parameters was derived for each spectral 
region with high absorption. Ranges with minor atmospheric ab¬ 
sorption were corrected with molecfit using the standard pa¬ 
rameters given in Table 3. To achieve a comparison based on op¬ 
timised conditions for both methods, the classical method was 
also applied to each spectral piece individually. In addition, the 
positions of the base points for the continuum fit were individu- 

14 http://iraf.net/irafhelp.php?val=telluric&help= 
Help+Page 


Table 11. Table of individual regions used for the telluric absorption 
correction. 


region 

# 

wavelength 
range [/mi] 

included 

molecules 

a 

0.9402-1.1079 

O", CO", H 2 0*, CH", CO" 

b 

1.1687- 1.2537 

O", CO", H 2 0*. CH", CO" 

c 

1.2800-1.3023 

O", CO", H 2 O fc , CH", CO" 

d 

1.5159- 1.7610 

O", CO", H 2 O fc , CH", CO" 

e 

1.9827-1.9918 

O", CO", H 2 0*, CH", CO" 

f 

2.0333-2.0435 

O", CO", H 2 O fc , CH", CO" 

g 

2.0809-2.2400 

O", CO", H 2 O fc , CH", CO" 

h 

2.3210-2.3500 

O a T CO", H 2 O fc , CH", CO" 

1 

1.1079-1.1687 

H^O* 

2 

1.2537- 1.2800 

6/ 

3 

1.3023- 1.5159 

H 2 O b 

4 

1.7610-1.9827 

h 2 o*, co 2 b 

5 

1.9918-2.0333 

h 2 o*. co 2 b 

6 

2.0435-2.0809 

h 2 o*. co 2 b 

7 

2.2400-2.3210 

ch/ 

8 

2.3500-2.3600 

CH/ 


Notes. 

(o ' molecular abundance calculated, but not fitted ,w molecular abun¬ 
dance fitted 


ally chosen for every TSS spectrum to obtain an optimal trans¬ 
mission curve. 

5.2. Results 

The comparison of the quality of the telluric absorption correc¬ 
tion achieved with the classical TSS and molecfit method re¬ 
veals that in some regions with minor atmospheric absorption 
both methods perform similarly, e.g. the weak C0 2 bands be¬ 
tween 1.6 and 1.615/zm are well corrected (see Fig. 17). There 
are also some regions with major atmospheric absorption, where 
both methods achieve very good absorption correction (e.g. red- 
wards of about 2 fim in Fig. 18). 

However, usually noticeable differences in the quality of the 
correction are visible. These differences even become critical for 
regions #1 through #8, where major absorption bands affect any 
ground based observation. In total, we identified three classes of 
critical problems: 

5.2.1. Object continuum reconstruction 

Reconstructing the continuum of the science target in a reliable 
way is a difficult issue, particularly in regions with broad absorp¬ 
tion regions like #3 and #4 (see Fig. 16 and Table 11). The qual¬ 
ity for the continuum reconstruction with this implementation of 
the classical method crucially depends on the incorporated inter¬ 
polation. As such, a fit only can be based on a limited number of 
base points. Artificial continuum variations are unavoidably in¬ 
troduced in any wavelength range, even with minor or no molec¬ 
ular absorption. Fig. 19 shows an example of a poor continuum 
reconstruction in a minor absorption region, where the Brackett 
series and some Fell lines of the B(e) star (Kraus et al. 2012) 
arise (see Fig. 19a). The continuum is not well reproduced in 
the region around the Brackett line (4-11), even though this re¬ 
gion is only marginally affected by absorption. Another exam- 
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pie is shown in Fig. 18, where the spectrum of the EO galaxy 
NGC5812 is given in the range of a prominent CO 2 absorption 
region. The continuum shortwards of about 2 pm derived with 
the TSS method is too low, leading to an overcorrection of the 
continuum. 

5.2.2. Line correction/reconstruction 

Reliable reconstruction of object lines can be a difficult matter 
for the telluric absorption correction. It becomes critical if a line 
intrinsic to the TSS affects a scientifically important line in the 
science spectrum. For example, this applies to hydrogen lines, 
which is demonstrated by a spectrum of the PN IC1266. Fig. 20 
shows the hydrogen Pa^ line at 1.2818 pm, which has signifi¬ 
cantly higher flux and much broader wings when corrected with 
the classical method. This is induced by the fitting method to de¬ 
rive the transmission curve, which shows significant differences 
compared with the one derived with molecfit (see lower panel 
of Fig. 20). In addition, a small wavelength shift is visible in 
Fig. 20 that is probably caused by a small radial velocity shift 
affecting the TSS Hip085885. 

Another example is given in Fig. 21, which shows a spec¬ 
trum of the EO galaxy NGC5812 with significant overcorrec¬ 
tion by the classical method arising from TSS Brackett lines. As 
molecfit incorporates a purely theoretical transmission curve, 
such intrinsic stellar features do not occur. 

One of the possible drawbacks of using molecfit could be 
the sensitivity of the fit to intrinsic object lines, if they have not 
been excluded. To investigate this, we used the spectrum of data 
set #4, the elliptical galaxy NGC5812, which shows some in¬ 
trinsic features in region #5 between 1.995 and 2.035 pm (see 
upper panel of Fig. 18). This region is dominated by carbon 
dioxide and minor water vapour absorption. We only used the 
fitting range #5 (between 1.9918 and 2.0333 pm), where H 2 O 
and CO 2 were varied. It can be seen that the intrinsic object fea¬ 
tures remain unchanged (see Fig. 18). To estimate how sensi¬ 
tive molecfit is with respect to object lines, we again fitted 
region #5, but excluded the object lines from the fit (see blue 
areas in upper panel of Fig. 18). We find only marginal differ¬ 
ences in the telluric absorption corrected spectra of +20 ppm. 
We therefore consider molecfit to be robust with respect to 
single object lines, at least if object features do not dominate the 
corresponding fitting range. However, care should be taken when 
prominent intrinsic spectral object features or even entire bands 
are expected at the same wavelength as strong telluric absorp¬ 
tion lines. These features might be visible e.g. in low mass stars, 
molecular clouds, or planetary atmospheres. Prominent features 
indeed might influence the fit by mimicking differing molecular 
abundances in the Earth’s atmosphere. Hence, the contributions 
of the astronomical target and the Earth’s atmosphere cannot be 
disentangled by molecfit. In this case, the user is advised to 
either mask these features, if applicable, or choose different fit¬ 
ting ranges to avoid an unintended removal of object features. 
Alternatively, if such features are known to be visible in the en¬ 
tire spectrum, molecfit can be applied to a corresponding TSS 
and subsequent usage of the resulting transmission spectrum for 
the correction of the science target. 

5.2.3. Increased noise 

The finite S/N of the TSS becomes critical when it is smaller 
or comparable to the one of the object spectrum. Applying a 
noisy transmission for the telluric correction derived by means 



wavelength form] 

Fig. 17. Comparison of the telluric absorption correction methods: un¬ 
corrected object spectrum (black), molecfit corrected spectrum (red 
line), and the spectrum corrected with the classical method (green) in a 
region with minor atmospheric absorption (CO 2 band). 

of such a spectrum unavoidably degrades the S /N of the sci¬ 
ence spectrum. Fig. 22 shows a spectral region with negligible 
atmospheric absorption, where both methods usually lead to a 
good reproduction of intrinsic spectral object features (except 
the variations at 1.094 pm introduced by the TSS transmission). 
Since the theoretical transmission curve obtained with the radia¬ 
tive transfer code does not show any random noise, molecfit 
does not introduce noise-driven features in the science spectrum. 
However, the classical method does change the noise level. This 
becomes more and more significant, the more prominent the 
atmospheric absorption features are. For example, the promi¬ 
nent bands from 1.3 to 1.5 pm (H 2 O) and from 1.8 to 2.0 pm 
(H 2 O and CO 2 ) lead to very noisy science spectra, when the 
telluric absorption correction is done with the classical method 
(see Figs. 23 and 24, respectively). This is particularly important 
when object features of scientific interest are located there. 

6. Summary and conclusion 

We have developed the software package molecfit consisting 
of routines to fit synthetic atmospheric transmission spectra to 
science data (see Paper I) and to apply these synthetic spectra 
as telluric absorption correction to science files. We have exten¬ 
sively tested the software with a large X-Shooter data set to eval¬ 
uate the performance of the package with two figures of merit, 
the offset and the small-scale residual parameters, 7 c ff and 7 res , 
respectively. Moreover, we compared the telluric correction by 
molecfit with the classical method based on TSS for several 
science spectra. In the following, we summarise our findings: 

- The telluric absorption correction with molecfit of TSS 
does not introduce systematic offsets in the corrected spectra. 
The scatter of 7 0 ff is about 3% of the line strength. The rela¬ 
tive small-scale residual strength is about 20% (see Sect. 4.3 
and Fig. 5) for the NIR arm, i.e. the quality of the correction 
is roughly proportional to the strength of the telluric absorp¬ 
tion lines. The VIS arm data show results of similar quality. 

- The telluric correction shows a dependency on the number of 
pixels per FWHM. The small-scale residuals increase with 
decreasing slit width or FWHM (see Sects. 4.3, 4.4, and Ta¬ 
ble 6) if the pixel size is kept constant. 

- The quality of the fit crucially relies on the selection of the 
fitting ranges. In particular, all critical molecular absorp¬ 
tion bands existing in the wavelength range of the science 


Article number, page 14 of 17 



















W. Kausch et al.: Molecfit: A general tool for telluric absorption correction 



1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 

wavelength [pm] 


Fig. 16. Upper panel: Telluric standard star spectrum Hip084409, test data set #3, see Tables 1 and 2) and the fitting method for extracting a 
transmission curve: A cubic spline fit to base points (blue dots in upper panel) is used to derive the continuum of the telluric standard star. This fit 
is then used to normalise the spectrum to achieve the corresponding transmission (lower panel). We divided the spectrum into several wavelength 
regions based on the amount of absorption. Regions #1 through #8 are heavily affected by absorption, in contrast to regions (a) through (h) (see 
Table 11). Each region was corrected individually with molecfit and IRAK Middle panel: The resulting transmission curve including some 
molecular absorption features. Lower panel: Theoretical transmission curve achieved with molecfit. 
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Fig. 19. Panel (a) shows the Brackett series from V921-Sco, a B(e) star. Panel (b) is a zoomed in region with bad continuum correction by the 
classical method (colour coding as in Fig. 17). 


spectrum should be covered by corresponding regions (see 
Sect. 4.5). On the other hand, a few narrow fitting ranges are 
sufficient to achieve a good quality of the correction over the 
entire wavelength range. Spreading these narrow fitting re¬ 
gions over the entire wavelength range helps to improve the 
fit for the line spread function and the wavelength calibration 
correction. 


- Molecfit offers the possibility to influence the fit by ad¬ 
justable parameters. The selection of appropriate parameters 
is crucial for a good result. In particular, the degree of free¬ 
dom for the fit should the minimised, e.g. the Chebyshev 
polynomial for the wavelength grid correction has a great 
influence on the fitting quality. This should be chosen care¬ 
fully if the fitting ranges are small compared to the entire 
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Fig. 18. Upper panel: Example for poor continuum reproduction by the 
classical method (colour coding as in Fig. 17). Although the prominent 
CO 2 absorption feature is well corrected by both methods, significant 
variations in the continuum are visible in the spectrum corrected by 
the classical method. Intrinsic object lines redwards of 2 pm are well 
reconstructed by both methods. Lower panel: relative residuals of the 
corrected spectra when object lines are excluded from the fit in region 
#5 (blue areas in upper panel). 



1.28 1.2805 1.281 1.2815 1.282 1.2825 1.283 1.2835 1.284 
wavelength Dun] 

Fig. 20. Pa^ line visible in the spectrum of the PNIC1266. Upper panel: 
Although the line is located in a wavelength range with minor atmo¬ 
spheric absorption, it is not well corrected with the classical method. An 
intrinsic wavelength shift is probably introduced by the radial velocity 
of the object (colour coding as in Fig. 17). Lower panel: The transmis¬ 
sion derived with the classical method (green line) shows a broad dip 
leading to a significant higher flux and different line shape in the cor¬ 
rected spectrum. The transmission derived with molecfit (red) does 
not show this feature. 



Fig. 21. Example of an overcorrection by spectral features arising from 
the Brackett lines (H(4-10). H(4-ll), and H(4-12)) visible in the TSS. 
In the molecfit based correction such features do not occur (colour 
coding as in Fig. 17). 



1.084 1.086 1.088 1.09 1.092 1.094 1.096 

wavelength [gm] 

Fig. 22. Upper panel: Detailed zoomed in region with minor atmo¬ 
spheric absorption. Due to variations caused by the spline fit and noise 
in the TSS spectrum, the classical method introduces continuum vari¬ 
ations and noise to the corrected spectrum. All spectral features in the 
theoretical transmission arise from molecular absorption. Since it is free 
from random variations, the noise in the corrected spectrum is not in¬ 
creased (colour coding as in Fig. 17). 


wavelength range covered by the spectrum as in the case of 
X-Shooter (see Sect. 4.6). 

- Comparisons with atmospheric profiles measured by a mi¬ 
crowave radiometer reveal that the fitting algorithm is very 
robust with respect to variations of the input atmospheric 
profile. Thus, the initial profile does not have a large effect as 
long as it does not deviate extremely from the real one. The 
PWV value determined with molecfit tends to be too high 
in the case of very dry observing conditions. However, this 
does not affect the quality of the telluric corrections. 

- Molecfit is also applicable to low S/N data. However, there 
may be a loss of quality in the telluric correction for very 
low S/N observations leading to more residuals. Data with 
extremely low S /N cannot be fitted reliably. In this case, a 
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Fig. 23. Close up of the prominent water vapour absorption feature 
between 1.3 and 1.5 /jm. Due to the intrinsic noise of the TSS spec¬ 
trum and the subsequent telluric absorption correction, the noise intro¬ 
duced by the classical method is significantly higher (colour coding as 
in Fig. 17). 



wavelength Qam] 

Fig. 24. Same as in Fig. 23 but for the prominent water vapour and car¬ 
bon dioxide absorption feature between 1.8 and 2.0/um (colour coding 
as in Fig. 17). 


TSS can alternatively be used for the fit instead of the science 
spectrum (see Sect. 4.8). 

- We performed a comparison with the classical method, 
which is affected by the following problems: 

- (a) The implementation of the TSS continuum determi¬ 
nation can lead to significant continuum changes in the 
corrected spectrum, e.g. by the limited number of fitting 
points, and subsequent normalisation of the spectrum. 

- (b) Intrinsic spectral lines of the TSS can change the flux 
and shape of object lines or mimic additional spectral 
features in the corrected object spectrum. 

- (c) The intrinsic noise of the TSS observation lead to ad¬ 
ditional noise in the corrected object spectrum. 

Whenever such a TSS-related problem appears, it can be 
expected that molecfit performs significantly better, as 
demonstrated in several examples. Moreover, direct fitting 
of the science spectrum with molecfit avoids issues related 
to differences in the atmospheric conditions for the science 
and TSS observations. 

- Molecfit is very robust with respect to single object lines 
lying in the fitting regions, at least if the regions are not dom¬ 
inated by these features. The telluric correction is only af¬ 
fected marginally in this case. However, care should be taken 


when molecular bands are expected in the scientific target, 
which are present in the fitting ranges. 

The incorporation of synthetic atmospheric transmission spec¬ 
tra based on theoretical calculations provide a promising way 
to perform the telluric absorption correction. The highly effi¬ 
cient underlying algorithm of molecfit offers the opportunity 
to achieve a reasonable and reliable correction without supple¬ 
mentary observations of TSS which are time expensive. In ad¬ 
dition, the applicability of molecfit with standard parameters 
allows already reasonable results. In addition, optimising these 
standard parameters can be achieved in much less time than op¬ 
timising the standard method, making molecfit a very efficient 
tool for the telluric absorption correction. We also successfully 
applied molecfit to several other ESO instruments covering 
several wavelength and resolution regimes (see Paper I). Only 
a slight adaption of instrument dependent parameters is neces¬ 
sary. Any required information can be added in the parameter 
file if the FITS header of the files does not contain ESO compli¬ 
ant keywords, which are read by molecfit. Although the soft¬ 
ware is delivered with meteorological data for Cerro Paranal, it 
provides the capability to create atmospheric profiles appropriate 
also for other observing sites. This flexibility makes molecfit 
a general tool for the telluric absorption correction adaptable to 
various instruments and observing sites. 
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